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Summary 

Human  factors  research  into  spatial  disorientation  (SD)  and  loss  of  situation  awareness  (SA)  in  the  fast  jet 
military  pilot  has  led  to  consideration  of  systems  for  monitoring  pilot  behaviour  and  psychophysiology  and  for 
detection  of  performance  degradation  and  incapacitation.  These  could  be  the  basis  of  real-time 
countermeasures,  such  as  a “Safety  Net”  system,  assisting  or  taking  over  automatic  control  until  the  pilot  is 
able  to  resume  full  control  of  the  system.  This  paper  looks  at  technologies  developed  under  the  UK  MOD 
“Cognitive  Cockpit”  project  for  providing  cognitive  assistance  through  adaptive  automation  and  decision 
support.  The  paper  considers  the  requirements  for  monitoring  and  countermeasures  for  cancelling  SD.  It  is 
argued  that  all  three  basic  types  of  SD  can  be  cancelled  by  effective  real-time  adaptive  countermeasures  using 
flexible  levels  of  autonomy  governed  by  pilot  agreed  plans.  Through  analysis  and  cognitive  walk-through  of  a 
mission  story-board,  we  show  how  the  safety  net  concept  can  be  extended  by  cognitive  automation  to  provide 
augmentation  of  SA  and  decision  making.  Cognitive  augmentation  can  be  seen  to  mitigate  the  most  dangerous 
form  of  insidious  disorientation,  by  keeping  the  pilot  in  the  control  loop  before  SD  sets  in. 

1.  INTRODUCTION 

Since  the  late  1980’s,  researchers  in  aviation  human  factors  and  medicine  have  been  concerned  about  the 
effects  of  high  mental  workload  and  physiological  stresses  on  the  operators  of  fast-jet  fighter  aircraft, 
particularly  the  consequences  of  G-induced  loss  of  consciousness  (G-LOC),  spatial  disorientation  (SD)  and 
loss  of  situation  awareness  (SA).  This  led  to  consideration  of  the  development  of  systems  for  detection  of 
performance  degradation  and  pilot  incapacitation,  and  the  potentially  controversial  concept  of  a “Safety  Net” 
system,  temporarily  overriding  the  authority  of  the  partially  or  fully  incapacitated  pilot  until  he/she  is  able  to 
resume  full  control  of  the  system.  Generally,  operators  expressed  guarded  acceptance  of  the  safety  net 
concept,  with  concerns  about  system  reliability  and  tactical  utility  of  operator  monitoring,  advisory  and 
recovery  systems  in  an  operational  mission  context.  The  need  to  override  the  authority  of  the  pilot  proved 
hard  to  sell. 

Subsequently,  many  of  the  sub-components  of  the  safety  net  system  concept  for  providing  significant  risk 
mitigation  have  been  successfully  developed  and  implemented  in  operational  systems.  These  include: 

1)  systems  for  automatic  G-protection  in  aircrew  life-support  equipment, 

2)  improved  sensor  and  computing  technologies  providing  information  fusion,  situation  assessment,  mission 
management  and  decision  support, 

3)  advanced  control/display  technologies  for  improved  communication  interfaces,  and 

4)  new  automation  techniques  for  aircraft  control  and  limitation  of  operation  and  performance,  such  as 
manoeuvre  envelope  protection,  and  aircraft/ground  proximity  warning  and  collision  avoidance  systems. 


Paper  presented  at  the  RTO  HFM  Symposium  on  “ Spatial  Disorientation  in  Military  Vehicles: 

Causes,  Consequences  and  Cures”,  held  in  La  Coruna,  Spain,  15-17  April  2002,  and  published  in  RTO-MP-086. 
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However,  full  implementation  of  the  safety  net  concept  is  yet  to  be  achieved.  In  particular,  there  remains  a 
need  to  develop  sufficiently  reliable,  trustworthy  systems  for  the  monitoring  and  detection  of  operator 
performance  degradation  (e.g.  G-LOC,  SD,  loss  of  SA),  that  can  trigger  credible,  effective  and  tactically 
useful  interventions,  preferably  without  the  need  to  override  the  pilot's  authority. 

Under  the  UK  Ministry  of  Defence  (MOD)  Applied  Research  Program,  a three-year  program  of  human 
sciences  led  work  on  intelligent  pilot  aiding  has  been  recently  completed  at  DERA  Centre  for  Human 
Sciences,  Farnborough.  This  MOD  “Cognitive  Cockpit”  (COGP1T)  project  successfully  demonstrated  the 
feasibility  of  providing  automated  cognitive  assistance  to  aid  the  fast-jet  pilot  through  coupling  adaptive 
automation  and  decision  support  concepts  with  technologies  for  monitoring  operator  behaviour  and 
psychophysiology  (1).  This  application  of  cognitive  automation  focussed  on  aiding  in  a high  cognitive  load 
scenario,  with  possible  distraction  and  reduced  SA,  involving  the  use  of  defensive  aids  and  mission  replanning 
in  response  to  a pop-up  threat.  A key  project  goal  was  to  enhance  system  adaptiveness  in  response  to 
dynamically  changing,  time  pressured  mission  environments.  Achieving  this  needed  development  of  detailed 
understanding  of  the  operator  requirements  for  control  and  management  of  automated  and  semi-automated 
tasks.  It  was  found  to  be  essential  to  provide  a system  for  task  management  that  retains  the  operator’s 
authority  and  executive  control  of  critical  system  functions,  whilst  enabling  delegation  of  responsibility  to  the 
computer  for  the  performance  of  tasks  as  required.  This  would  require  an  appropriate  balance  of  feed-forward 
and  feedback  information  on  task  performance.  The  key  to  this  problem  was  two-fold: 

1)  The  provision  of  systems  for  monitoring  and  reasoning  with  contexts  (pilot  state,  mission,  and 
environment)  with  a high  degree  of  context  sensitivity  i.e.  both  precision  and  accuracy. 

2)  The  development  of  a simple,  readily  understandable  and  easily  controllable  set  of  flexible  levels  of 
autonomy  and  cognitive  assistance,  short  of  full  automation,  with  an  intuitive  cognitive  interface  enabling 
effective  cognitive  interaction. 

These  requirements  are  believed  to  be  central  to  the  provision  of  a credible,  trustworthy  safety  net  system, 
which  can  successfully  intervene  under  G-LOC,  SD  or  loss  of  SA. 

The  purpose  of  this  paper  is  to  summarise  the  work  and  lessons  learnt  relevant  to  the  safety  net  concept,  with 
particular  reference  to  the  use  of  flexible  levels  of  autonomy  and  the  reliability  of  advanced  techniques  for 
operator  functional  state  assessment.  The  aim  is  to  describe  the  relevance  of  the  system  concepts  for  cognitive 
augmentation  mitigating  against  loss  of  SA  and  the  onset  of  SD. 

2.  SITUATION  AWARENESS  AND  SPATIAL  DISORIENT AION 

2.1  Definitions  and  types 

SA  has  been  defined  as  “the  perception  of  the  elements  in  the  environment  within  a volume  of  time  and  space, 
the  comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the  near  future”  (2).  Applied  to  the 
military  aviation  environment,  SA  would  include  elements  such  as  navigation,  tactics,  communications, 
weather,  aircraft  capability,  and  spatial  orientation  (3).  It  is  possible  for  a pilot  to  lose  situational  awareness.  In 
one  study,  the  majority  of  SA  errors  occurred  as  the  result  of  failure  to  perceive  the  situation;  that  is,  failure  to 
detect  information  or  misperception  of  information  (4).  It  follows  that  failure  to  perceive  one’s  spatial 
orientation  can  result  in  SD,  defined  as  an  incorrect  sense  of  the  position,  motion  or  attitude  of  the  aircraft  or 
of  the  pilot  within  the  fixed  co-ordinate  system  provided  by  the  surface  of  the  earth  and  the  gravitational 
vertical  (5). 

The  mechanism  of  SD  is  complex,  involving  the  interaction  of  the  vestibular  and  somatosensory  senses  in 
conjunction  with  the  visual  axis.  Recognition  of  SD  and  subsequent  recovery  requires  the  cognition  of  the 
pilot.  Traditionally  SD  has  been  categorised  as  Type  I,  when  the  pilot  is  unaware  that  he  is  disoriented;  Type 
II,  he  is  aware  that  he  is  disoriented;  and  Type  III,  he  is  aware  but  is  unable  to  overcome  the  disorientation. 
Type  I SD  is  insidiously  dangerous,  whereas  Type  II  and  III  SD  are  obviously  dangerous.  Clearly  the  most 
dangerous  condition  is  that  of  Type  1 SD. 

In  a study  of  USAF  mishaps  from  1991-2000,  spatial  disorientation  accounted  for  20.2  % of  all  Class  A 
mishaps,  costing  over  $1.4  billion  and  60  fatalities.  The  four  most  common  contributors  to  SD  mishaps  in 
decreasing  order  of  frequency  were  attention  management,  judgement/decision  making,  mission  demands, 
and  psychological  factors  (6).  Because  most  of  the  contributors  involve  cognitive  tasks,  it  should  be  apparent 
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that  an  aircraft  system  that  augments  the  cognitive  ability  of  the  pilot  would  decrease  or  eliminate  the 
consequences  of  SD. 

2.2  Real-time  countermeasures 

Initial  applications  of  cognitive  automation  have  focussed  on  aiding  in  a mission  scenario  with  high  cognitive 
load,  distraction  and  high  potential  for  reduced  SA.  A high  level  of  SA  and  spatial  orientation  is  required  in 
military  aviation  in  order  to  prosecute  the  mission,  especially  in  single  seat  fast  jets.  With  the  increased 
technological  ability  of  multiple  role  fighters  and  the  capability  of  deploying  anywhere  in  the  world,  any 
enhancement  of  a pilot’s  SA  could  provide  the  difference  between  success  and  failure.  Today’s  pilots  must 
cope  with  multifunction  displays,  multiple  sensor  sources,  various  audio  inputs,  avionic  helmets,  and  real  time 
information  flowing  in  and  out  of  the  cockpit.  Simultaneously  the  pilot  must  keep  a running  mental  model  of 
the  mission  and  electronic  order  of  battle. 

In  a way,  this  is  no  less  true  for  the  pilot  of  Unmanned  Aerial  Vehicles  (UAV);  in  fact,  enhancement  of  SA 
may  be  even  more  crucial  since  the  UAV  pilot  is  remote  from  the  vehicle  and,  to  some  degree,  out  of  the  loop. 
Providing  real-time  feedback  of  the  state  of  the  UAV,  the  mission,  and  the  cognitive  load  on  the  pilot  is 
crucial  to  mission  success.  Manned  or  unmanned,  the  pilot  cannot  become  spatially  disoriented,  losing  the 
relationship  of  the  aircraft  to  the  earth.  The  consequence  can  be  controlled  (or  even  uncontrolled)  flight  into 
terrain.  And  in  fighter  pilot  lingo,  “a  kill  is  a kill.” 

Cognitive  automation  has  the  capability  to  cancel  the  consequences  of  SD.  Constantly  monitoring  the  pilot’s 
functional  state,  the  aircraft  and  its  environment,  and  the  execution  of  the  mission  plan  provides  the  platform 
for  maintaining  the  highest  degree  of  SA.  The  pilot  cannot  be  caught  in  Type  I SD,  which  essentially  obviates 
the  likelihood  of  controlled  flight  into  terrain  (CFIT).  Even  if  the  pilot  does  experience  Type  II  SD,  such  as 
the  leans,  he  either  flies  the  aircraft  with  reference  to  instruments,  or  allows  the  aircraft  to  fly  itself,  based  on 
the  level  of  automaticity.  In  Type  III  SD,  the  cognitive  automation  should  recognise  that  the  pilot  is  not 
functioning,  and  the  computer  will  take  control  until  the  pilot  has  recovered.  If  the  cognitive  automation 
prevents  SD,  USAF  data  show  that  $1.4  billion  could  be  saved  and  60  fatalities  prevented. 

3.  COGNITIVE  AUTOMATION 

3.1  Cognitive  Automation  Tasks 

Cognitive  automation  provides  an  intelligent  assistant  to  the  fast  jet  pilot.  This  assistant  has  two  main  goals:  it 
must  prevent  the  pilot  from  being  overloaded,  by  doing  things  that  he  would  otherwise  be  doing;  and  it  must 
help  the  pilot  by  providing  him  with  the  information  he  needs,  when  he  needs  it.  These  goals  break  down  into 
eight  tasks  (1): 

• MONITORING  PILOT:  watching  the  pilot  to  see  what  he  is  doing  and  how  he  is  coping  with  flying  the 
mission 

• MONITORING  ENVIRONMENT:  watching  the  outside  world,  the  aircraft  systems  and  incoming 
information  to  identify  anything  that  the  pilot  needs  to  know  or  do  something  about 

• MONITORING  MISSION  PLAN:  keeping  track  of  what  the  pilot  is  doing  and  what  he  needs  to  do  in  the 
near  future 

• REPLANNING  MISSION:  working  out  what  should  be  done  by  the  pilot  in  response  to  unexpected 
situations 

• UPDATING  MISSION  PLAN:  keeping  the  mission  plan  consistent  and  up  to  date 

• CONFIGURING  COCKPIT:  making  sure  that  the  pilot  has  the  information  he  needs  in  the  form  he  needs 
it,  and  that  the  controls  he  is  using  are  configured  to  let  him  do  the  things  he  wants  to  do  quickly  and 
easily 

• DECIDING  AUTOMATION:  working  out  what  tasks  should  be  automated  rather  than  left  to  the  pilot 

• AUTOMATING  TASKS:  actually  carrying  out  various  tasks  for  the  pilot 
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A simplified  model  of  the  overall  process  within  the  designed  system  is  shown  in  the  activity  diagram  in 
Figure  1.  Cognitive  automation  monitors  three  aspects  of  the  situation: 

• the  environment,  which  includes  the  outside  world  and  the  state  of  the  aircraft  systems; 

• the  mission  plan,  to  indicate  which  tasks  the  pilot  is  currently  engaged  in  and  what  he  will  be  doing 
shortly; 

• and  the  pilot,  to  take  into  account  his  cognitive  state. 

These  feed  forward  information  into  the  replanning  of  the  mission,  the  automation  of  tasks  that  have 
previously  been  identified  as  requiring  automation  and  configuring  the  cockpit  to  supply  the  pilot  with 
information  relevant  to  the  task  he  is  doing.  The  results  of  monitoring  the  pilot,  replanning  the  mission, 
automating  tasks  and  deciding  automation  cause  updates  to  the  mission  plan. 

The  performance  of  these  activities  is  a continual  process  throughout  the  mission.  The  various  aspects  of  the 
situation  are  continually  monitored,  and  may  result  in  changes  in  the  use  of  displays  and  other  output  devices, 
in  the  actions  recommended,  and  in  the  automation  level  adopted. 


Figure  1 . Cognitive  automation  tasks  and  information  flows 

3.2  Cognitive  Automation  Systems 

Three  classes  of  cognitive  automation  sub-systems  or  modules  are  needed  to  perform  the  goals  and  tasks 
identified  above. 

Situation  Assessment  Support  System  (SASS)  - This  is  a module  that  monitors  the  status  of  the  aircraft 
situation  and  the  outside  environment  and  recommends  actions.  This  module  is  concerned  with  on-line 
mission  analysis,  aiding  and  support  provided  by  real-time,  multi-agent  Knowledge  Based  Systems  (KBS) 
software.  This  system  is  privy  to  the  current  mission,  aircraft  (e.g.  heading,  altitude  and  threat)  and 
environmental  status,  and  is  also  invested  with  extensive  a priori  tactical,  operational  and  situational 
knowledge.  Overall,  this  system  provides  information  about  the  objective  state  of  the  aircraft  within  a mission 
context,  and  uses  extensive  KBS  to  aid  and  support  pilot  decisions  (7).  Knowledge  acquisition  (KA)  under  the 
COGP1T  programme  has  focussed  on  the  following  areas: 

• plan  assessment  - checking  how  the  mission  is  progressing, 

• system  health  - checking  how  the  aircraft  systems  are  performing, 

• attack  phase  - carrying  out  the  attack  on  the  target, 

• and  Defensive  Aids  System  (DAS)  and  re-routing  - identifying  when  DAS  and  re-routing  should  be 
employed  to  counter  threats  and  weather. 
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Knowledge-level  models  of  these  tasks  have  been  built,  and  decision  support  has  been  implemented  and 
successfully  demonstrated  for  DAS  re-routing  in  a high  cognitive  load  scenario,  with  the  potential  for  loss  of 
SA.  The  SASS  decision  support  provided  the  basis  for  augmented  cognition  and  maintained  SA.  Further  KA 
from  subject  matter  experts  is  needed  to  build  the  knowledge  base  for  implementing  the  correct  decision 
support  in  tactical  scenarios  involving  reduced  pilot  capacity  and  pilot  incapacitation  (loss  of  SA,  SD,  G- 
LOC).  In  an  SD  scenario,  SASS  would  receive  pertinent  aircraft  and  environmental  information  (e.g.  rapid 
accelerations,  high  G manoeuvres;  unusual  aircraft  attitudes,  positions,  flight  path  trajectories;  unusual  rates  of 
climb/descent,  loss  of  altitude  and  energy;  closure  on  obstacles,  terrain).  SASS  would  provide 
recommendations  on  appropriate  timings  and  actions  plans  for  tactically  correct  aircraft  and  mission  control 
(e.g.  automated  flight  control,  re-routing,  threat  countermeasures,  and  terrain  collision  avoidance). 

Cognition  Monitor  (COGMON)  - This  is  a module  that  monitors  the  pilot’s  physiology  and  behaviour  to 
provide  an  estimation  of  pilot  state.  This  module  is  concerned  with  on-line  analysis  of  the  psychological, 
physiological  and  behavioural  state  of  the  pilot.  Primary  system  functions  include  continuous  monitoring  of 
workload,  and  inferences  about  current  attentional  focus,  ongoing  cognition  and  intentions.  It  also  seeks  to 
detect  dangerously  high  and  low  levels  of  arousal.  Overall,  this  system  provides  information  about  the 
objective  and  subjective  state  of  the  pilot  within  a mission  context.  This  information  is  used  in  order  to 
optimise  pilot  performance  and  safety,  and  provides  a basis  for  the  implementation  of  pilot  aiding  (8).  In  an 
SD  scenario,  COGMON  would  receive  relevant  pilot  information,  estimate  the  cognitive  affective  state  (e.g. 
detecting  and  identifying  G-LOC  onset,  attentional  focussing,  overload,  fatigue,  drowsiness),  and  provide 
information  on  the  pilot’s  ability  to  perform  tasks  (e.g.  alerting,  full  automation). 

Tasking  Interface  Manager  (TIM)  - This  is  a module  that  monitors  the  mission  plan  and  manages  the  interface 
with  which  the  pilot  is  presented.  This  module  is  concerned  with  on-line  analysis  of  higher-order  outputs  from 
COGMON  and  SASS,  and  other  aircraft  systems.  A central  function  for  this  system  is  maximisation  of  the 
goodness  of  fit  between  aircraft  status,  ‘pilot-state’  and  tactical  assessments  provided  by  the  SASS.  These 
integrative  functions  enable  this  system  to  influence  the  prioritisation  of  tasks  and,  at  a logical  level,  to 
determine  the  means  by  which  pilot  information  is  communicated.  Overall,  this  system  allows  pilots  to 
manage  their  interaction  with  the  cockpit  automation,  by  context-sensitive  control  over  the  allocation  of  tasks 
to  the  automated  systems  (9).  The  TIM  functional  architecture  comprised  modules  for  goal -plan  tracking  and 
for  interface,  timeline,  automation  and  task  management  utilising  a blackboard  for  goal-plan  tracking 
information.  In  a SD  scenario,  TIM  would  receive  SASS  and  COGMON  information,  identify  conflicts  and 
discrepancies,  and  integrate  these  with  information  on  the  mission  plan  tasks  and  goals.  TIM  would  manage 
the  implications  for  the  performance  of  tasks  (e.g.  non-adherence  to  plan,  non-responsiveness  to  planned 
changes),  assist  or  automate  re-planning  and  task  performance  as  required,  and  provide  the  pilot  with 
appropriate  feed-forward  and  feedback  control  information. 


4.  MONITORING  THE  PILOT’S  FUNCTIONAL  STATE 

The  COGMON  sub-system  has  a specific  role  within  the  cognitive  automation  system  aimed  at  identifying  the 
cognitive  affective  state  of  pilots,  and  may  be  seen  as  a means  of  identifying  how  well  the  pilot  is  coping  with 
current  task  demands.  This  can  be  used  to  identify  means  of  adaptively  automating  tasks  within  the  cockpit 
environment.  The  COGMON  uses  four  classes  of  information  to  estimate  the  cognitive  affective  state  (Figure 
2).  These  are  physiological  measures,  behavioural  measures,  subjective  measures  and  contextual  measures.  It 
is  the  key  interactions  between  each  of  these  classes  of  measures  that  allow  predictions  to  be  made  of  the  type 
of  cognitive  operation  being  performed.  When  this  is  combined  with  a subjective  assessment  of  the  workload 
experienced  by  individuals  when  performing  that  specific  task,  and  the  physiological  correlates  of  high 
workload,  for  example,  then  estimates  of  an  individuals  ability  to  cope  with  current  task  demands  may  be 
made. 

Advanced  flight  systems  also  give  access  to  the  wealth  of  information  regarding  the  local  environment  in 
which  the  pilot  is  operating.  For  example,  orientation  of  aircraft,  threat  status,  G experienced  by  pilot,  altitude 
etc.  This  provides  a context  in  which  our  physiological,  behavioural  and  subjective  measures  can  be 
interpreted.  This  information  has  to  be  more  than  merely  of  academic  interest  if  the  problems  presented  by 
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spatial  disorientation  are  to  be  mediated.  The  output  from  the  COGMON  are  designed  to  provide  a useful 
index  of  the  pilots  status,  and  should  enable  principles  of  adaptive  automation  to  be  employed  for  the  safe  re- 
orientation of  the  pilot  and  aircraft. 


Figure  2.  Structure  of  COGMON  measures  and  state  estimates 

The  spatial  disorientation  literature  presents  a picture  in  which  conflicting  visual,  vestibular  and 
proprioceptive  information  leads  to  increased  disorientation.  This  is  exacerbated  by  the  requirement  for  the 
‘high  level’  forms  of  cognitive  processing,  such  as  those  associated  with  strategic  decision  making,  and 
attention  management.  Indeed  it  has  been  reported  that  combining  unambiguous  sensory  information  leads  to 
reduced  levels  of  perceived  workload  during  the  maintenance  of  situational  and  spatial  awareness.  The 
mechanism  of  operation  of  the  COGMON  is  metaphorically  illustrated  below. 

The  situation  whereby  the  attitude  of  the  aircraft  is  nose-down,  and  accelerating,  results  in  a shear  force  being 
present  on  the  otolith  organs  of  the  inner  ear.  This  combined  with  the  relational  movement  of  the  direction  of 
gravity,  combines  to  present  a perceived  horizontal  flight  path.  This  illusion  can  be  easily  disambiguated  if  the 
pilot  is  ‘head  out’  and  can  refer  to  the  out  of  the  window  (OTW)  reference  frame.  However  if  operational 
factors  require  the  pilot  to  be  head  down,  performing  complex  tasks,  such  as  interacting  with  a digital  map 
display,  or  calculating  re-routing  options,  then  the  OTW  frame  of  reference  cannot  be  accessed.  There  is  a 
growing  body  of  evidence  which  suggest  that  increased  levels  of  workload  are  a contributing  factor  to 
accidents  involving  SD.  Furthermore  if  the  task  requires  a high  degree  of  attentional  tunnelling,  the  ability  to 
process  external  stimuli  and  ‘task  share’  will  be  further  diminished.  Thus  monitoring  aircraft  systems  will 
enable  the  nose-down  attitude  and  the  increase  in  air-speed  to  be  identified.  The  interaction  of  the  pilot  with 
the  map  display  could  indicate  that  focus  of  attention  is  head-down,  and  that  attention  is  being  invested  in 
visuo-spatial  information  processing.  This  will  also  serve  to  limit  the  ability  of  the  pilot  to  react  to  additional 
visuo-spatial  information,  and  may  serve  to  compound  the  disorientation  instigated  by  the  illusion  of 
horizontal  flight.  Furthermore  the  analysis  of  EEG  recorded  over  posterior  and  parietal  scalp,  those  areas 
which  overlie  the  areas  of  the  brain  in  which  visuo-spatial  information  is  processed,  shows  high  levels  of 
coherence  between  activity  recorded  at  parietal  and  occipital  scalp  sites.  This  type  of  activity  may  indicate 
high  levels  of  visuo-spatial  workload.  This  example  demonstrates  that  available  information  may  be  integrated 
to  develop  a picture  of  the  cognitive  affective  state  of  the  pilot  in  which  their  ability  to  respond  to  external 
stimuli  is  diminished  due  to  current  task  demands. 
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5.  AUTOMATION  AND  CONTROL  OF  TASKS 

5.1  Tasking  Interfaces 

The  idea  of  a tasking  interface  exploited  the  lessons  learnt  from  the  US  Army’s  RPA  program  (10).  It  arose 
from  the  need  to  be  able  to  predict  pilot  expectations  and  intentions  with  reference  to  embedded  knowledge  of 
mission  plans  and  goals.  The  aim  was  to  provide  an  adaptive  or  “tasking”  interface  that  allowed  the 
operators/pilots  to  pose  a task  for  automation  in  the  same  way  that  they  would  task  another  skilled 
crewmember.  It  afforded  pilots  the  ability  to  retain  executive  control  of  tasks  whilst  delegating  their 
execution  to  the  automation.  A tasking  interface  necessitated  the  development  of  a cockpit  interface  that 
allowed  the  pilot  to  change  the  level  of  automation  in  accordance  with  mission  situation,  pilot  requirements 
and/or  pilot  capabilities.  It  was  necessary  that  both  the  pilot  and  the  system  operated  from  a shared  task 
model,  affording  communication  of  tasking  instructions  in  the  form  of  desired  goals,  tasks,  partial  plans  or 
constraints  that  were  in  accord  with  the  task  structures  defined  in  the  shared  task  model. 

Allowing  pilots  to  choose  various  levels  of  interaction  for  the  tasks  they  are  required  to  conduct  can  mitigate 
the  problem  of  unpredictability  of  automation.  The  TIM  can  utilise  the  monitoring  and  analysis  of  the  mission 
tasks  provided  by  the  SASS  combined  with  the  pilot  state  monitoring  of  the  COGMON  to  afford  adaptive 
automation,  adaptive  information  presentation  and  task  and  timeline  management. 

5.2  Contractual  Autonomy 

Providing  flexible  levels  of  autonomy  for  the  performance  of  tasks  and  functions  is  a key  requirement  for 
implementation  of  the  tasking  interface  concept.  An  important  development  under  the  COGPIT  project  was 
the  framework  devised  for  providing  only  the  necessary  and  sufficient  levels  of  autonomy  with  TIM  support. 
The  resultant  framework  is  known  as  the  system  for  pilot  authorisation  and  control  of  tasks,  or  PACT, 
described  in  Table  1(11,  12). 


Primary 

Modes 

Levels 

Operational 

Relationship 

Computer 

Autonomy 

Pilot 

Authority 

Adaptation 

Information  on 
performance 

AUTOMATIC 

Automatic 

Full 

Interrupt 

Computer 
monitored  by 
pilot 

On/off 

Failure  warnings 
Performance  only  if 
required. 

ASSISTED 

4 

Direct 

Support 

Advised 

action 

unless 

revoked 

Revoking 

action 

Computer 
backed  up  by 
pilot 

Feedback  on  action. 
Alerts  and  warnings  on 
failure  of  action. 

3 

In  Support 

Advice,  and 
if 

authorised, 

action 

Acceptance 
of  advice 
and 

authorising 

action 

Pilot  backed 
up  by  the 
computer 

Feed-forward  advice  and 
feedback  on  action.  Alerts 
and  warnings  on  failure  of 
authorised  action. 

2 

Advisory 

Advice 

Acceptance 
of  advice 

Feed-forward  advice 

1 

At  Call 

Advice  only 
if  requested. 

Full 

Pilot,  assisted 
by  computer 
only  when 
requested. 

Feed-forward  advice,  only 
on  request 

COMMANDED 

Under 

Command 

None 

Full 

Pilot 

None  performance  is 
transparent. 

Table  1.  Bonner-Taylor  PACT  framework  for  pilot  authorisation  and  control  of  tasks 

PACT  is  based  on  the  idea  of  contractual  autonomy.  Borrowing  an  aircrew  term  from  co-operative  air 
defence,  the  idea  is  that  the  pilot  forms  a contract,  or  set  of  contracts,  with  the  automation  using  the  PACT 
system  by  allocating  tasks  to  PACT  modes  and  levels  of  automation  aiding.  The  contract  defines  the  nature  of 
the  operational  relationship  between  the  pilot  and  the  computer  aiding  during  co-operative  performance  of 
functions  and  tasks.  Autonomy  is  limited  by  a set  of  contracts,  or  binding  agreements,  made  between  the  pilot 
and  the  computer  automation  system  governing  and  bounding  the  performance  of  tasks.  Through  PACT 
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contracts,  the  pilot  retains  authority  and  executive  control,  while  delegating  responsibility  for  the  performance 
of  tasks  to  the  computer. 

5.3  Control  of  PACT 

PACT  succeeds  in  reducing  the  number  of  automation  or  autonomy  modes  required  to  three  - namely,  fully 
automatic,  assisted  or  pilot  commanded  - with  a further  four  secondary  levels  nested  within  the  semi- 
automatic, assisted  mode,  which  can  be  changed  adaptively  or  by  pilot  command.  The  PACT  system  uses 
military  terminology  for  categories  of  support  for  Army  land  forces  military  operations  (At  call,  advisory,  in 
support,  direct  support)  to  afford  usability  and  compatibility  with  military  user  cognitive  schemata  and 
models.  It  provides  realistic  operational  relationships  for  a logical,  practical  set  of  levels  of  automation, 
reduced  to  six  levels  of  autonomy,  with  progressive  operator/pilot  authority  and  computer  autonomy 
supporting  situation  assessment,  decision  making  and  action  (Figure  3).  Mission  functions  and  tasks,  at 
different  levels  of  abstraction  allocated  individually  or  grouped  in  related  scripts  or  plays,  can  be  set  to  these 
levels  in  a number  of  ways: 

• Pre-set  operator  preferred  defaults, 

• Operator  selection  during  pre-flight  planning, 

• Changed  by  the  operator  during  in-flight  re-planning,  probably  using  Hands  on  Throttle  and  Stick 
(HOTAS),  touch  screen,  and  Direct  Voice  Input  (DVI)  commands  (1 1). 

• Automatically  changed  according  to  operator  agreed,  context-sensitive  adaptive  rules. 


AUTOMATION 

PACT 


PILOT 

AUTHORITY 


Interrupt 


Revoking 

action 


Acceptance  of  advice 
& authorising  action 


Acceptance 
of  advice 


Advice,  only 
if  requested 


Pilot  full 
authority 


COMPUTER 

AUTONOMY 


Full  computer 
autonomy 


Advised  action, 
unless  revoked 


Advice,  and  if 
authorised,  action 


Advice 


Advice,  only 
if  requested 


Pilot  full 
authority 


Figure  3.  PACT  levels  of  pilot  authority  and  contractual  autonomy 


5.4  Intervention  Strategy 

An  overview  of  the  proposed  adaptive  intervention  strategy  is  illustrated  in  Figure  4.  In  SD  scenarios,  pilot 
awareness  is  variable  and  unpredictable.  Thus,  it  is  important  that  the  PACT  level  changes  can  be  triggered 
adaptively  in  response  to  contextual  input  from  COGMON,  SASS  and  TIM  mission  goal-plan  tracking  (GPT), 
carried  out  in  accordance  with  the  pilot’s  pre-set  PACT  contracts.  COGMON,  SASS  and  TIM  will  contribute 
different  triggering  information,  depending  on  the  type  of  SD  involved.  SASS  will  be  important  for  Type  1 SD 
with  only  weak  COGMON  inferences  (e.g.  distraction,  inattention).  There  should  be  strong  SASS,  COGMON 
and  TIM  indications  for  Type  II  and  III  SD.  Table  2 summarises  the  kinds  of  indications  and  inferences  that 
could  be  available  from  monitoring  the  pilot,  the  environment  and  the  mission  plan  in  Type  I,  II  and  III  SD 
scenarios. 
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AUTOMATION 

PACT 


DIRECT  SUPPORT 


ADVISORY 


AT  CALL 


COMMANDED 


COGNITIVE  STATE 
& MISSION  PERFORMANCE 

Monitoring  & management  of  performance  variability 


25 

m 

s 

iS 

OPT  SASS 


Monitor,  detect,  reflect,  correct 


TIM 

SALIENCY 


COMPELLING 


INFLUENCING 


BACKGROUND 


Figure  4.  PACT  adaptive  intervention 

The  intervention  strategy  seeks  to  monitor  and  manage  the  variability  in  performance  through  a barrier  system 
approach  (monitor,  detect,  correct,  reflect),  and  through  appropriate  cognitive  stream  interventions  (join, 
break,  divert).  TIM  feedback  and  feed-forward  control  messages  are  provided  to  the  pilot  with  appropriate 
multi-modal  intervention  saliency  (background,  hinting,  influencing,  directing,  compelling)  developed  to 
manage  attentional  and  cognitive  bias  (13).  The  ability  to  manage  the  saliency  TIM  messages  should  be 
particularly  useful  for  communicating  adaptive  countermeasure  information  in  SD  scenarios  with  complex 
sensory  and  cognitive  demands.  Different  intervention  saliency  requirements  will  apply  when  the  pilot  is 
distracted  and  unaware  of  the  problem  (Type  I SD),  compared  with  when  aware  of  conflicting  sensory  inputs 
and  needing  to  focus  (Type  II  SD),  or  when  unable  to  overcome  compelling  SD  (Type  III  SD).  Figure  5 
illustrates  how  the  prototype  cockpit  display  formats  were  developed  to  communicate  TIM  feed-forward  and 
feedback  control  information  with  managed  saliency  in  the  Head  Mounted  Display  (HMD)  and  Head  Down 
Display  (HDD).  Auditory  voice  messages,  with  directional  sound,  provided  powerful  augmentation  of  the 
visual  information. 

As  discussed  earlier,  tasks  can  be  pie-allocated  to  possible  PACT  level  contracts  by  the  pilot  before  the  flight 
in  mission  planning.  The  individual  task  PACT  levels  (defaults  and  contingencies)  should  be  set  to  mitigate 
the  risks  to  achievement  of  the  individual  task  goals.  The  TIM  Task  Manager  distinguishes  between  pending, 
active  and  completed  tasks  for  the  current  mission  element.  Individual  tasks  progress  from  pending,  to  active 
and  to  complete  as  the  mission  progresses.  An  example  of  the  Task  Manager  status  is  highlighted  in  the  TIM 
control  station  display  developed  for  the  COGPIT  demonstration  (Figure  6).  The  panels  shown  in  Fig  6 are 
clockwise,  from  top  right,  as  follows: 

• lnferencing  Display  (list  of  goals  and  tasks,  indicating  currently  active) 

• COGMON  display  - load  estimates  (green  = low;  yellow  = medium;  red  = high)  of  pilot  states  (visual, 
verbal,  auditory,  spatial,  left  hand,  right  hand,  motor,  alertness,  tactile) 

• TIM  Task  Display  - pending,  active  and  completed  tasks,  with  associated  active  or  default  and 
contingency  PACT  contracts,  indicating  the  current  and  permitted  alternative  PACT  levels. 

• COGM  AN  Display  - visualisation  of  visual,  verbal,  auditory,  left  hand,  right  hand  loads,  and  location  of 
gaze 
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TASK 

DESCRIPTION 

SD  SCENARIO 

TYPE  I 
Unaware 
Not  responding 

TYPE  II 
Aware 

Trying  to  cope 

TYPE  III 

Aware-Unable  to 
overcome 

MONITORING 

PILOT: 

watching  the  pilot  to 
see  what  he  is  doing 
and  how  he  is  coping 
with  flying  the 
mission 

Weak  COGMON 

indications 

Behavioural: 

Head-down,  focussed  not 
OTW  referenced. 

Normal,  unresponsive 
actions  e.g.  map  gaze 
location,  no  compensatory 
flight  control  activity, 
normal  speech  comms. 
Psychophy  siological : 
Normal  relaxed  EEG, 
ECG,GSR,  DC  shifts,  low 
HR,  HR  variability,  blink. 

No  clear  inference  re  in  or 
out  of  the  loop  control,  or  of 
intervention  need. 

Strong  COGMON 

indications 

Behavioural: 

Reactive  e.g.  visual  focussing 
on  instruments,  compensatory 
flight  control  inputs,  reduced 
speech. 

Psycho-physiological: 

Raised  alertness,  high  load, 
raised  stress,  focussed 
attention  e.g.  alert  EEG,  ECG, 
GSR,  raised  HR,  HR 
variability,  reduced  blink 
duration,  frequency. 

Pilot  in  the  loop,  but 
assistance  may  be 
appreciated. 

Clear  COGMON 

indications 

Behavioural: 

Scrambled  or  emergency 
procedures,  speech 
communication 
Psycho-physiological  - High 
stress,  cognitive  overload,  sensory 
conflict  EEG,  ECG,  GSR,  HR, 
HRV,  blink,  gaze,  nystagmus 

G-LOC:  inaction,  no  speech  and 
psycho-physiological  correlates 

Evidence  of  pilot  out  of  the  loop. 
Assistance  almost  certainly 
required. 

MONITORING 
ENVIRONMENT: 
watching  the  outside 
world,  the  aircraft 
systems  and  incoming 
information  to 
identify  anything  that 
the  pilot  needs  to 
know  or  do  something 
about 

Strong  SASS 
indications 

Unexpected  changes,  drifts, 
trends  in  aircraft  attitude, 
altitude,  position,  speed, 
acceleration,  velocity 
vector,  flight  path. 
Orientation  towards  terrain, 
obstacles,  threats. 

Early  external  indications  of 
possible  need  for  cautionary 
warning,  awareness. 

Clear  SASS  indications 
Significant  departures, 
unusual  aircraft  attitude, 
altitude,  position,  speed, 
acceleration,  velocity  vector, 
flight  path.  Approaching 
terrain,  obstacles,  threats. 
Delayed  or  incomplete 
countermeasures. 

N.B.  Does  not  always  apply. 
Type  II  can  be  associated 
with  normal  flight  parameters. 

Assistance  almost  certainly 
required. 

Clear  SASS  indications 
Extreme,  unsafe  aircraft  attitude, 
altitude,  position,  speed, 
acceleration,  velocity  vector, 
flight  path.  Immediate  terrain, 
obstacles,  threats.  None  or 
ineffective  countermeasure 
actions. 

Immediate  action  required. 

MONITORING 
MISSION  PLAN: 
keeping  track  of  what 
the  pilot  is  doing  and 
what  he  needs  to  do  in 
the  near  future 

Weak  TIM  inferences 
Current  task  still  as 
planned.  No  clear  indication 
that  the  pilot  intends  to  take 
a different  flight  trajectory, 
or  to  change  the  mission 
plan,  or  that  the  changes  are 
inadvertent  and  unintended. 
No  clear  inference  of 
intervention  need  from 
plan/task  tracking  alone. 
Discrepancies  between 
environment  and  pilot 
information  can  indicate 
potential  awareness 
problem  and  warning  need. 

Strong  TIM  inferences 
Recognised  task  change. 
Current  task  not  as  planned, 
nor  pending.  Identifiable  as 
priority  flight  control 
recovery  action.  Increased 
risk  to  mission  plan. 

Needs  assistance  with 
recovery  task  and  possibly  in 
assessing  the  consequences 
for  the  mission  plan. 

Clear  TIM  inferences 
Current  priority  flight  control 
recovery  task  not  being 
performed  effectively.  Pilot  safety 
and  mission  goals  at  extreme  risk. 

Needs  immediate  automatic 
recovery  of  flight  control  and 
probably  assistance  with  re- 
planning the  mission,  particularly 
if  incapacitated. 

Table  2 Monitoring  SD 


Feed-forward 

advice 


IR1995 


Auto-flcti uate  HR LB 
Option  Deploy  KALB 
About  to  Chaff 
About  to  Chaff 
Auto- Jam 


DIRCrt 


-Current- 


MR! 


Feed-forward 
advised  Q 
action 


COGMON 


Task  Default/active  Contingency 

type  PACT  level  PACT  level 

contract  downward 

v changes  \ 


Chaff  Release 


Completed 


Pending  task 


Figure  6 TIM  Control  station  display  of  tasks  and  PACT  levels 
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In  Figure  6,  the  left-most  panel  of  the  TIM  Task  Display  shows  the  most  recently  completed  task 
(“Jamming”),  whereas  the  active  (“Beam  ground  threat”)  and  pending  panels  (“Chaff  release”;  “Activate 
MALD”  etc)  show  multiple  tasks  as  necessary.  Individual  tasks  are  represented  as  boxes  containing  their 
name  together  with  their  associated  possible  default  and  contingency  PACT  level  contracts,  which  range  from 
0 (Commanded)  to  5 (Automatic).  The  “Chaff  Release”  task  is  highlighted  with  its  default  PACT  level  display 
set  for  5 (Automatic).  Two  additional  PACT  level  displays  are  available  to  the  right  of  this  default  display  to 
show  contingencies  for  how  the  PACT  level  can  change  by  increasing  (t)  or  reducing  (4-)  PACT  levels 
according  to  pre-set  contracts.  In  this  example,  when  the  default  PACT  level  is  set  at  5,  it  is  possible  under 
certain  circumstances  (e.g.  Chaff  remaining  < 30%)  for  the  PACT  level  to  change  down  to  4,  2 or  0 (but  not  3 
or  1).  Explanatory  information  on  the  circumstances  for  triggering  changes  (contract  details)  is  stored  and 
available  for  inspection.  PACT  levels  that  are  unavailable  are  also  indicated  (reverse  contrast  caption)  as 
shown  for  pending  Chaff  Release  task,  default  PACT  level  1 . 

6.  COGNITIVE  AUTOMATION  SD  COUNTERMEASURES 

In  SD  scenarios,  pilot  awareness  and  capacity  for  countermeasure  action  is  variable  and  unpredictable.  Table  2 
has  shown  the  indications  and  inferences  that  could  be  available  from  monitoring  the  pilot,  the  environment 
and  the  mission  plan  in  Type  1,  II  and  111  SD  scenarios.  In  the  proposed  cognitive  automation  control  system 
(Figure  1),  these  monitoring  functions  feed  forward  information  into  mission  re -planning,  the  automation  of 
tasks,  and  configuring  the  cockpit  with  relevant  task  information.  In  SD  scenarios,  these  control  tasks  can  be 
considered  as  the  constituents  of  real-time  cognitive  automation  SD  countermeasures.  Strong  environment 
(SASS),  pilot  state  (COGMON)  and  mission  plan  (TIM)  indications  are  likely  to  be  available  for  detecting 
Type  II  and  III  SD,  and  for  guiding  the  kinds  of  countermeasures  envisioned  under  the  safety  net  concept. 
This  includes  the  possibility  of  temporarily  automating  control  of  safety  critical  system  functions,  with 
implicit  or  explicit  pilot  consent,  in  the  case  of  the  partially  or  fully  incapacitated  pilot  until  he/she  is  able  to 
resume  full  control  of  the  system.  For  Type  1 SD,  the  availability  of  strong  and  sensitive  environment 
monitoring  information  for  comparison  with  pilot  state  and  mission  plan  information,  raises  the  possibility  of 
countermeasures  for  this  insidious  and  particularly  dangerous  form  of  SD,  and  is  believed  to  be  most 
significant.  Type  1 and  II  SD  countermeasures  seem  likely  to  take  the  form  of  augmenting  SA,  decision 
support  and  cognitive  assistance  i.e.  designed  to  keep  the  pilot  in  the  control  loop.  It  seems  that  in  considering 
Type  1 or  II  SD,  where  possible  cognitive  augmentation  and  assistance  that  keeps  the  pilot  in  the  control  loop, 
is  probably  always  preferable  to  substituting,  if  only  temporarily,  fully  automatic  control  systems  for  the  pilot. 
This  could  be  for  a number  of  good  reasons  associated  with  risks  of  automation  control,  including  loosing  the 
tactical  benefits  of  pilot  in  the  loop  SA,  the  inherent  unpredictability  of  automation,  automation  mode 
awareness,  delayed  pilot  recovery  of  control,  automation  bias,  overuse  and  trust,  and  pilot  skill  fade. 

Table  3 summarises  the  implications  for  Type  I,  II  and  III  SD  countermeasures  of  the  cognitive  automation 
tasks  of  re-planning  the  mission,  updating  the  mission  plan,  configuring  the  cockpit,  deciding  automation,  and 
automating  tasks.  It  seems  clear  that  there  are  significantly  different  implications  for  re-planning,  cockpit 
configuration  task  management  and  automation,  cognitive  augmentation  with  the  SD  types. 

• Re-planning  and  Updating  the  Mission  Plan.  Type  I SD  indications  are  unlikely  to  warrant  immediate 
re-planning  and  task  changes,  but  might  usefully  trigger  cautionary  warnings  and  advice  to  confirm  the 
mission  plan  and  task.  However,  TYPE  II  and  Type  III  SD  indications  are  likely  to  warrant  checking  the 
mission  plan  and  flight  control  task.  Type  111  SD  indications  may  require  automatic  re-planning  including 
automation  of  mission  and  flight  control  tasks. 

• Configuring  the  Cockpit.  Type  I SD  indications  probably  warrant  the  need  for  alerting  cautionary  and 
warning  HMD,  HDD  and  audio  information  augmenting  SA,  with  intelligently  managed  saliency,  and 
advice  to  check  and  confirm  understanding  of  the  mission  and  flight  control  status.  Type  11  SD  indications 
probably  warrant  salient  feed-forward  orientation  and  recovery  information  using  most  effective  HMD, 
HDD,  audio  and  tactile  display  cueing  methods,  and  offer  of  automatic  recovery  options.  Type  111  SD 
indications  probably  require  provision  of  most  salient  feed-forward  HMD.  HDD  and  voice  information 
and  intuitive  controls  (HOTAS,  DVI)  for  automation  recovery  options,  and  availability  of  pilot  state 
feedback. 
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TASK 

DESCRIPTION 

[ SD  SCENARIO  1 

TYPE  I 
Unaware 
Not  responding 

TYPEII 

Aware 

Trying  to  Cope 

TYPE  III 
Aware 

Unable  to  overcome 

REPLANNING 
MISSION: 
working  out  what 
should  be  done  by  the 
pilot  in  response  to 
unexpected  situations 

No  clear  indication  of 
immediate  re-planning  need. 
Early  external  indications 
from  S ASS  of  need  for 
cautionary  warning,  and 
possible  need  to  confirm  the 
mission  plan. 

Inferred  immediate  priority  task 
and  goal  change  (safe  recovery) 
with  possible  need  to 
automatically  check,  and  advise 
any  implications  for  the  mission 
plan. 

Automatically  assess  any 
implications  of  pilot  (e.g. 
incapacity)  and  aircraft  state  (e.g. 
position)  for  the  mission  plan  and 
ability  of  pilot  to  perform  tasks.  Re- 
plan to  include  automation  of  tasks 
as  appropriate. 

UPDATING 
MISSION  PLAN: 
keeping  the  mission 
plan  consistent  and  up 
to  date 

No  immediate  update 

required. 

Update  mission  plan  if  required. 

Update  mission  plan  as  required. 

CONFIGURING 
COCKPIT: 
making  sure  that  the 
pilot  has  the 
information  he  needs 
in  the  form  he  needs  it, 
and  that  the  controls 
he  is  using  are 
configured  to  let  him 
do  the  things  he  wants 
to  do  quickly  and 
easily. 

• Head  Down 
Display  (HDD) 

• Helmet  Mounted 
Display  (HMD) 

• Direct  Voice 
Input  (DVI) 

• Hands  on  throttle 
and  stick 
(HOTAS) 

Provide  pilot  alert/warning 
information.  Indicate  need  to 
check  information  on  aircraft 
flight  control  and  mission 
plan  and  confirm 
understanding  of  status. 
Manage  level  of  TIM  f-fwd 
message  saliency. 

For  awareness:  TIM4  Hinting 
or  T1M3  Influencing. 

For  immediate  awareness: 
TIM2  Directing,  or  TIM  1 
Compelling. 

HMD  & HDD  and  voice  e.g. 
“Check  attitude”  or  “Pull-up, 
pull-up”. 

Support  f-fwd  orientation  and 
recovery  action  with  high 
saliency  cueing: 

TIM2  Directing;  TIM1 

Compelling 

Aiding  options  include: 

1)  status  HMD  (e.g.  NDFR), 
HDD  (e.g.  BAI),  3-D  audio, 
tactor. 

2)  Priority  recovery  action 
(e.g.  “Pull-up,  pull-up”;  HMD 
sky  pointers;  directional 
tactors). 

3)  Tactically  safe  manoeuvre 
(e.g.  voice  “Better  terrain 
right"). 

4)  Flight  path  director 
predictor  (pathway  in  the  sky). 

5)  Force-stick  tactile  cueing, 
+override. 

6)  Automatic  recovery  option 
(e.g.  “Option  auto  pilot”). 

Provide  information  and  controls 
for  automation  recovery  option. 
Provide  high  saliency  f-fwd 
information:  TIM  1 Compelling. 
HMD, HDD  & voice:  “Auto  avoid” 
“Brace,  brace”. 

Provide  HOTAS  manual  and  DVI 
auto  selection  and  over-ride 
options. 

Provide  pilot  state  indicator  control 
panel  (self  report). 

Provide  COGMON  state  estimation 
feedback  information  HMD,  HDD 
and  voice  (TIM3  Influencing)  e.g. 
“Reduced  pilot  capacity”. 

DECIDING 
AUTOMATION: 
working  out  what 
tasks  should  be 
automated  rather  than 
left  to  the  pilot. 

• Flight  Control 
System  (FCS) 

• Ground  Collision 
Avoidance 
System  (GCAS) 

• Mission  Control 
System  (MCS) 

• Defensive  Aids 
System  (DAS)  - 
Jamming, 
Manoeuvre 
Chaff,  MALD 

No  immediate  full  auto 
requirement. 

Pre-set  PACT  Assisted  mode 
level  changes  triggered  by 
events  (SASS): 

1)  FCS:  PACT1  At  Call  to 
PACT2  Advisory 

2)  GCAS:  PACT2  Advisory 
to  PACT3  In  Support 

3)  DAS:  PACT3  In  Support 

Pilot  selected,  or  pre-set  PACT 
Assisted  mode  level  changes 
triggered  by  events  (COGMON, 
SASS,  TIM): 

1)  FCS  Recovery:  PACT2 
Advisory  to  PACT3  In  Support 

2)  GCAS:  PACT3  In  Support  to 
PACT4  Direct  Support 

3)  MCS:  Re-planning  PACT1 
At  Call  to  PACT2  Advisory 

4)  DAS:  PACT3  In  Support  to 
PACT4  Direct  Support 

Pre-set  PACT  Assisted  mode  level 
changes  triggered  by  events 
(COGMON,  SASS,  TIM): 

1)  FCS  Recovery':  PACT3  In 
Support  to  PACT4  Direct  Support 

2)  GCAS:  PACT4  Direct  Support 

3)  MCS:  Replanning  PACT2 
Advisory  to  PACT4  Direct  Support 

4)  DAS:  PACT4  Direct  Support 

AUTOMATING 

TASKS 

actually  carrying  out 
various  tasks  for  the 
pilot 

Alerting  & advice. 

FCS  PACT2  Advisory 
GCAS  PACT3  In  Support 

Advice  and  action,  if  required. 
FCS  PACT3  In  Support 
GCAS  PACT4  Direct  Support 
MCS  PACT2  Advisory 

Advised  action,  unless  revoked. 
FCS  PACT4  Direct  Support 
GCAS  PACT4  Direct  Support 
MCS  PACT4  Direct  Support 
DAS:  PACT  4 Direct  Support 

Table  3 SD  Countermeasures 
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• Deciding  Automation  and  Automating  Tasks.  Type  1 SD  indications  probably  warrant  adaptive  Assisted 
PACT  level  changes  to  provide  automatic  Flight  Control  System  (FCS)  alerting  and  advice  (PACT  2 
Advisory),  and  raised  Ground  Collision  Avoidance  System  (GCAS)  support  for  action  (PACT  3 In 
Support).  Type  II  SD  indications  probably  warrant  further  raising  of  adaptive  Assisted  PACT  levels  to 
provide  support  for  both  advice  and  action  (e.g.  FCS  : PACT  3 In  Support;  GCAS:  PACT4  Direct 
Support;  DAS:  PACT  4 Direct  Support)  and  raised  Mission  Control  System  (MCS)  Re-planning  (PACT  2 
Advisory).  Type  111  SD  indications  probably  warrant  raising  automation  level  of  all  mission  and  safety 
critical  systems,  including  FCS  recovery  and  MCS,  to  provide  action,  unless  revoked  (PACT  4 Direct 
Support). 

7.  STORYBOARD  COGNITIVE  WALKTHROUGH 

In  order  to  illustrate  the  operation  of  the  proposed  prototype  cognitive  automation  system,  a mission 
storyboard,  route  map  and  time-line  is  reported  as  an  annex  to  this  paper.  This  is  a modification  to  the  DAS 
Re-route  storyboard  used  by  DERA  CHS  for  the  successful  MoD  Final  Customer  Demonstration,  ARP26f2.3 
Automated  Decision  Support  assignment,  held  at  Farnborough  on  23  March  2001.  The  scenario,  events  and 
activities  are  believed  to  be  illustrative  of  a difficult  mission  with  high  levels  of  workload  and  stress.  The  level 
of  mission  difficulty  is  designed  to  defeat  a pilot  without  adaptive  aiding,  cognitive  automation  and  decision 
support.  The  story-board  events  and  action  timeline,  partially  illustrated  in  Figures  8,  9 and  10,  were  flown 
successfully  and  repeatedly  executed  by  Sqn  Ldr  Phil  O'Dell  RAF,  Fast-jet  Project  Test  Pilot,  DERA 
Boscombe  Down.  The  storyboard  builds  a typical  insidious  Type  I SD  event  in  the  context  of  a deep 
interdiction  attack  mission.  This  shows  the  operation  of  the  PACT  system  for  DAS  and  Re-routing  automation 
as  developed  and  demonstrated  by  DERA  CHS.  In  summary,  after  executing  a successful  target  attack  in  a 
threat  environment,  during  the  subsequent  egress  mission  segment,  the  pilot  encounters  2 pop-up  SAM 
threats,  which  are  countered  using  the  automated  DAS  actions.  Fusing  of  the  2nd  SA8  close  aboard  leads  to  a 
fuel  leak,  pressurisation  failure  and  fuel  falling  below  chicken  requiring  urgent  computer-assisted  re-routing  to 
safe  airspace  and  recovery  to  a diversion  airfield.  This  is  followed  by  COGMON  and  TIM  inferring  reduced 
pilot  capacity  further  triggering  of  changes  to  DAS  automation  levels.  In  the  final  segment  illustrated  in 
Figure  1 1,  a Type  I SD  event  is  added  to  the  closing  reduced  pilot  capacity  scenario.  COGMON  indications  of 
stress,  and  TIM  tracking  of  pilot  tasks  and  actions.  This  describes  adaptive  cognitive  automation  of  SD 
countermeasures  operating  in  accordance  with  the  capabilities  and  constraints  discussed  above,  and  identified 
in  Tables  2 and  3.  Appropriate  monitoring  and  countermeasures  for  Type  II  SD  and  Type  II  SD  are  illustrated 
in  Figures  12  and  13. 

8.  CONCLUSIONS 

SD  can  be  detected  from  monitoring  environmental,  behavioural  and  psycho-physiological  indicators.  It 
should  be  possible,  in  the  near  future,  to  implement  systems  that  use  this  information  together  with  embedded 
task  models,  to  identify  and  to  discriminate  Type  I,  II,  and  II  SD  states.  We  have  described  how  using  this 
monitoring  information,  SD  can  be  cancelled  by  effective  real-time  adaptive  countermeasures,  in  ways  that 
are  trustworthy  and  that  follow  provably  correct  pilot  agreed  plans.  An  important  step  has  been  the 
development  and  successful  proof-of-concept  demonstration  of  an  intuitive,  easily  controllable  set  of  flexible 
levels  of  autonomy  for  cognitive  assistance  and  automation.  This  enables  the  cognitive  interaction  needed  for 
maintaining  pilot  authority  and  delegating  responsibility  to  automation  for  the  performance  of  tasks.  Through 
the  use  of  flexible  levels  of  autonomy,  the  safety  net  concept  can  be  extended  to  provide  cognitive 
augmentation  that  mitigates  against  loss  of  SA  before  SD  sets  in,  keeping  the  pilot  in  the  control  loop. 
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ANNEX 

DAS/RE-ROUTE  SD  STORYBOARD 

The  scenario  is  for  a 2015-2020  singleton  FireFox  aircraft,  carrier  borne  option,  on  deep  interdiction,  day  low- 
level,  weather  dependent  mission  against  a target  airfield,  with  airborne  refuel,  SWEEP  and  SEAD,  and  with 
Fulchrum,  SA8  and  SA  14  mission  threats.  The  pilot  is  Sqn  Ldr  Mark  “Dell”  Cafferky-Seares.  The  intelligent 
aiding  system  (TIM  + SASS  & COGMON)  is  nick-named  “ODIN”  after  the  Norse  God  of  wisdom.  The 
story-board  segment  begins  in  the  egress  phase  of  the  mission,  some  5 minutes  after  the  successful  airfield 
attack,  as  indicated  on  the  route  plan  (Figure  7). 


Figure  7 Route  Plan  for  Attack  and  Egress 

Timeline:  16:06:30  - 16:08:00  (map  minutes) 

Profile:  Egress  route  flown  at  100-300  AGL  at  450kt. 

Sensors  in  use:  Radar  (TF  in  a GPWS  role  and  air-to-air),  RWR,  FLIR,  MAW 

Local  Weather:  5/8,h  cloud  at  5000ft.  Visibility  10  km.  Some  stratus  on  hills  above  8000ft.  Surface  wind  320/15. 

Pilotage:  Manual 

Route  position:  Wpt  22  to  23 

Pre-SD  Segment 

16:06:30  RWR  indicates  SAM  contact.  ODIN  advises  Dell  possible  Integrated  Air  Defence  System,  followed  by 
automated  DAS  action  (PACT3  jamming,  PACT2  cued  beam  manoeuvre,  PACT5>4  chaff,  PACT3  MALD  release, 
PACT5  stop  jamming).  The  time-line  in  Figure  8 illustrates  the  external  events  and  cockpit  messages  in  this  segment  up 
to  the  PACT  4 deployment  of  chaff. 
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Figure  8.  Time-line  SA8  TWS  to  PACT  4 Chaff  Deployed. 

16:06:39  ODIN  advises  Chaff  failure  and  Dell  manual  recycle  of  chaff  system  power.  MAW  indicates  missile  inbound, 
followed  by  ODIN  automated  PACT5  flares.  With  chaff  still  inoperative,  ODIN  (advised  by  COGMON)  detects  DELL's 
inattention  to  missile  (gaze  head-down  focussed  on  chaff  re-activation),  and  provides  repeated  audio  alert  “Missile  left, 
unknown’-  (TIM1)  and  HDD/HMD  cueing  leading  to  Dell  DVI  “Acknowledged”.  ODIN  PACT5  auto-evade  is 
interrupted  by  Dell’s  manual  last  ditch  manoeuvre  at  impact  minus  5.  ODIN  DVO  provides  MAW  count  down  “Missile 
impact  in  6, 5, 4, 3, 2,1”  followed  by  D1RCM  activity  fusing  missile  at  200  metres. 

16:06:46  RWR  indicates  SAM  radar  lock  followed  by  ODIN  DAS  action  (PACT4  jamming,  PACT2  cued  140  break 
lock  manoeuvre).  RWR  indicates  SAM  launch,  then  MAW  ODIN  DVO  counts  down  “Missile  impact  in  6, 5, 4, 3,2,1” 
followed  by  Dell  last  ditch  manoeuvre  and  DIRCM  activity  fusing  missile  close  aboard,  causing  airframe  buffeting. 

16:07:10  SAM  fire  control  radar  ceases.  ODIN  PACT5  system  check  detects  and  advises  fuel  leak,  followed  by 
pressurisation  failure,  restricting  ceiling  to  20,000  feet,  followed  by  cross-feed  failure,  restricting  fuel  below  chicken. 

The  fuel  loss  triggers  re-route  calculations  by  ODIN  (advised  by  SASS),  which  factors  the  fuel  level  of  below  chicken 
into  urgent  re-route.  ODIN  provides  advice  to  Dell  in  HMD  and  HDD  “Calculating  re-route”.  Dell  acknowledges  this 
information,  through  DVI  “Acknowledged”. 

SD  SEGMENT 

16:07:30  Key  Decision  Point:  Method  of  recovery  to  safe  airspace 
Task:  Recover  aircraft  to  diversion  field 

ODIN  provides  advice  to  Dell  in  HMD  and  HDD  to  clear  aircraft  “Consider  jettison  drop  tanks”. 

Task:  Release  drop-tanks 

There  is  no  immediate  response  from  Dell  to  this  ODIN  advice. 

Re-route  level  for  egress  is  set  in  the  mission  plan  at  PACT  2 (Advice  only).  ODIN  (advised  by  SASS)  provides  the 
information  to  allow  presentation  of  proposed  routes  on  the  digital  map  and  advises  two  routes  to  Dell  recommending 
route  A as  preference.  Dell  requests  more  information  on  the  two  routes  by  DVI  “ More  info  route  A”  then  “Head  down 
3 more  info  route  B”. 


27- 1.8 


Time  (s) 


Event 

Fuel 

Pilot  Request 

Pilot  Request 

below  Chicken 

More  Info  "A" 

HDD  3 More  1 

Acknowledge 


5"  time  out  then  3“ 


PLAN 

TIM  1 
TIM  2 


SAgS;  Calculate  Reroute,  Jettison  Drop  Tanks,  Stipulate  Recovery  Profile 


t I 

"Fuel  Below  Chicken" 


TIM  Feedback 
TIM  3-5 


▼ * 

“Calculating  "Consider  Jettison 

Reroute"  Drop  Tanks" 


▼ 

"2  Routes,  SASS 
suggests  Rte  A" 


t t 

Text  Details  of  Text  Details  of 

route  choice  route  choice 
and  MAP  and  MAP 


Figure  9.  Time-line  Chicken  Fuel  to  Re-route  assessment. 

ODIN  provides  information  on  each  route  in  the  HDD  in  sequence  Route  A followed  by  Route  B,  allowing  Dell  to  toggle 
between  maps. 

Dell  accepts  advice  for  high  level  transit  direct  to  diversion.  Dell  requests  specific  details  on  re-route  A using  DVI 
“Head  down  3 details  route  A” 

ODIN  provides  the  detail  on  re-route  A to  Dell  in  HDD  “Route  A Climb  to  20  000  ft  Heading  200  deg  Prepare  TRD’\ 

Task:  Gain  altitude  and  traverse  directly  to  diversion  field; 

Task:  Prepare  TRD; 

Task:  Monitor  fuel  supply 

Dell  requests  using  DVI  “More  information  pressurisation”.  ODIN  responds  HDD  “Max  ceiling  20000  feet  due  to 
pressurisation  failure". 

ODIN  (advised  by  COGMON)  interprets  that  Dell  is  suffering  some  degree  of  stress,  which  affects  Dell’s  state  and 
reduces  his  capability.  ODIN  (via  COGMON)  detects  that  Dell’s  gaze  remains  head-down  on  the  re-route  information, 
with  sustained  focussed  attention,  and  estimates  high  visuo-spatial  load. 

ODIN’s  previous  call  “Consider  release  drop  tanks”  has  produced  no  acknowledgement  and  no  action  response  from 
Dell.  Thus,  the  release  drop  tanks  task  remains  pending. 

There  is  no  advised  altitude  or  heading  change,  or  Towed  Radar  Decoy  (TRD)  preparation  action  in  response  to  the  re- 
routing advice.  These  tasks  also  remain  pending  (advised  by  TIM).  At  this  time,  ODIN  (advised  by  SASS)  notes  a loss  of 
altitude. 
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Figure  10.  Time-line  Re-route  advice  to  Pressurisation  query. 

ODIN  uses  this  information  combined  with  the  re-route  information  provided  by  SASS  to  change  the  level  of 
automation,  as  agreed  by  the  pilot  in  mission  planning,  and  to  provide  SD  countermeasures.  The  Flight  Control  System  is 
changed  from  PACT  1 (At  Call)  to  PACT  2(Advisory),  and  the  pilot  is  notified.  ODIN  advises  Dell  by  DVO  “Check 
altitude”.  Dell  hears  and  accepts  the  advice  by  DVI  “Acknowledged".  The  Ground  Collision  Avoidance  System  is 
changed  from  PACT  2 (Advisory)  to  PACT  3 (In  Support).  ODIN  advises  Dell  by  DVO  “Check  flight  plan”.  Dell 
accepts  the  advice  by  DVI  “Accept".  ODIN  (via  COGMON)  monitors  Dell  visually  checking  the  HDD  SASS  re-routing 
advice  for  Route  A.  Dell  then  initiates  the  climb  to  20000ft,  steers  to  heading  200,  and  initiates  deployment  of  the  TRD. 
ODIN  (via  TIM)  infers  the  pilot’s  intent  and  current  active  tasks  are:  Gain  altitude  and  traverse  directly  to  diversion 
field;  Prepare  TRD; 

As  agreed  by  the  pilot  in  mission  planning,  following  an  SD  event,  the  levels  of  the  remaining  ECM  countermeasures 
available  to  the  aircraft  are  increased,  as  the  re-route  will  cross  a SA-6  MEZ. 

The  ECM  automation  level  was  set  in  the  mission  plan  for  egress  at  PACT  3 (Advice  and  action  if  authorised).  ODIN 
assesses  the  pilot’s  capability  and  determines  that  it  should  be  increased  to  PACT  4 (advice  and  action  unless 
countermanded). 

Flare  automation  is  set  in  the  mission  plan  for  egress  at  PACT  3 (Advice  and  action  if  authorised).  ODIN  assesses  the 
Dell’s  capability  and  determines  that  it  should  be  increased  to  Level  4 (advice  and  action  unless  countermanded). 

Missile  avoidance  manoeuvre  automation  level  for  egress  is  set  in  the  mission  plan  at  PACT  3 (Advice  and  action  if 
authorised).  Given  the  fuel  restriction  ODIN  changes  this  automation  level  to  PACT  2 (Advice  only). 

The  DAS  PACT  information  is  automatically  displayed  on  a HDD  automation  page.  This  is  to  allow  Dell  to  query  and/or 
over-ride  any  of  the  adaptive  PACT  changes.  Dell  requests  information  on  these  changes  using  DVI  “ More  information 
PACT  change ” TIM  provides  advice  in  HMD  and  HDD  display  “Reduced  pilot  capacity  - SD  countermeasure”.  These 
monitoring  and  countermeasures  for  Type  I SD  are  illustrated  in  Figure  11;  possible  procedures  for  Types  II  and  III  SD 
are  illustrated  in  Figures  12  and  13. 
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Figure  11.  Timeline  SD  Type  I Countermeasures. 
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Figure  12.  Timeline  SD  Type  II  Countermeasures. 
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Figure  13.  Timeline  SD  Type  III  Countermeasures. 
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